feat(events): persist last processed ledger for crash recovery (#80) - #189

Closes #80

Problem Statement (The Bug)

The Stellar event poller (EventsService) only held its cursor — the last
processed ledger — in a private in-memory field:

private lastProcessedLedger = 0;

async onModuleInit() {
  const latest = await this.stellar.getLatestLedger();
  this.lastProcessedLedger = Math.max(1, latest - 10);
}

On every process restart (deploy, crash, OOM kill, pod reschedule) the
cursor was reinitialised to latestLedger - 10. Any contract event that
landed on-chain while the service was down — or more than 10 ledgers before
it came back — was never polled and never processed. Enrollment
safety-net updates, course_completed → certificate flows,
certificate_issued notifications and status syncs were silently dropped.

This cannot be fixed with a local patch because the failure is
architectural: the recovery point does not survive the thing it needs to
recover from. Widening the - 10 look-back only trades dropped events for
duplicate reprocessing and still has no lower bound on downtime. The cursor
has to live in durable storage that outlives the process.

Solution Comparison and Decision

Option D is the only approach that makes the recovery point as durable as the
data it protects.

The Change

New model (prisma/schema.prisma + migration
20260827000000_add_event_poller_checkpoint):

model EventPollerCheckpoint {
  id                       String    @id @default(uuid())
  key                      String    @unique
  lastProcessedLedger      Int       @default(0)
  lastPolledAt             DateTime?
  lastWriteError           String?
  consecutiveWriteFailures Int       @default(0)
  createdAt                DateTime  @default(now())
  updatedAt                DateTime  @updatedAt
  @@map("event_poller_checkpoints")
}

A single row keyed by stellar-event-poller.

New core method — persistCheckpoint() (best-effort, never throws):

private async persistCheckpoint(): Promise<void> {
  try {
    await this.prisma.eventPollerCheckpoint.upsert({
      where:  { key: EVENT_POLLER_CHECKPOINT_KEY },
      create: { key: EVENT_POLLER_CHECKPOINT_KEY, lastProcessedLedger: this.lastProcessedLedger, lastPolledAt: this.lastPolledAt },
      update: { lastProcessedLedger: this.lastProcessedLedger, lastPolledAt: this.lastPolledAt, lastWriteError: null, consecutiveWriteFailures: 0 },
    });
    this.consecutiveWriteFailures = 0;
    this.lastWriteError = null;
  } catch (error) {
    this.consecutiveWriteFailures += 1;
    this.lastWriteError = error?.message ?? String(error);
    this.logger.error(`Failed to persist ledger checkpoint (consecutive failure #${this.consecutiveWriteFailures}); polling continues`, this.lastWriteError);
    // record the failure on the row, but never let checkpoint I/O stop the poll loop
  }
}

pollEvents() now writes the checkpoint in a finally block — after a
successful batch, and also after an empty poll so the cursor keeps up during
quiet periods:

} finally {
  this.lastPolledAt = new Date();
  await this.persistCheckpoint();
}

onModuleInit() resumes from the checkpoint, with a guard against a
stale/foreign checkpoint:

Health endpoint (GET /health/checkpoint):

{
  "key": "stellar-event-poller",
  "lastProcessedLedger": 5123,
  "persistedLedger": 5123,
  "lastPolledAt": "2026-08-27T12:00:00.000Z",
  "resumedFromCheckpoint": true,
  "consecutiveWriteFailures": 0,
  "lastWriteError": null,
  "healthy": true
}

The aggregate GET /health gains an informational event_poller_checkpoint
indicator. It is deliberately non-fatal (always status: "up", carries a
checkpointHealthy flag) so a transient checkpoint write error never returns
a 503 that pulls the whole API out of the load balancer.

Compatibility Note

There is no INTERFACE_VERSION / API version constant in this codebase, so
nothing of that kind is bumped. Externally visible surface changes are
additive only:

• new table event_poller_checkpoints (new migration, no existing table touched)
• new route GET /health/checkpoint
• one new key inside the existing GET /health response

No existing request or response shape changes. EventsModule now exports
EventsService (consumed by HealthModule); no behavioural change to the
existing GET /events route.

Incidental Fixes

• Empty-poll cursor stall: the old if (!events.length) return; meant
  lastProcessedLedger was only ever touched when events existed. It now
  advances/persists every cycle.
• Silent RPC-failure init: a failed getLatestLedger() on boot used to
  drop straight to ledger 1 (full-history replay). It now prefers the
  persisted checkpoint and only falls back to 1 when there is genuinely no
  state.

Testing

New specs (all green):

src/modules/events/events.service.spec.ts

• resumes from the persisted ledger instead of the chain tip
• cold-starts near the chain tip and writes an initial checkpoint when none exists
• ignores a checkpoint that points past the chain tip (stale / wrong network)
• advances and persists the ledger after processing events
• still writes a checkpoint when a poll finds no events
• swallows the write error, records the failure and keeps advancing in memory
• recovers (failure counter resets) once a later write succeeds
• reports the persisted ledger from the database

src/modules/health/health.controller.spec.ts — resolves the controller
through a real TerminusModule so check() runs the actual
HealthCheckService aggregation and checkpoint() runs the real
GET /health/checkpoint handler:

• GET /health/checkpoint returns the current ledger checkpoint payload
• GET /health includes the event poller checkpoint indicator and stays 200 (ok)
• GET /health surfaces a degraded checkpoint without failing the overall check

$ npx jest src/modules/events/events.service.spec.ts src/modules/health/health.controller.spec.ts
PASS src/modules/events/events.service.spec.ts
PASS src/modules/health/health.controller.spec.ts
Test Suites: 2 passed, 2 total
Tests:       11 passed, 11 total

Adversarial case from the issue ("Manage checkpoint write failures without
blocking polling"): upsert is mocked to reject; pollEvents() still
resolves, consecutiveWriteFailures becomes 1, healthy becomes false,
and the in-memory cursor still advances to 111.

Before: no checkpoint persistence — FAILED to survive restart.
After: ok — cursor reloaded from Postgres on boot.

Full suite (npx jest) is green except for pre-existing, unrelated
TypeScript failures in src/modules/reviews/* and
src/modules/uploads/video-transcode.service.ts (references to Review /
ModerationLog models and a missing UploadResult export that are not in
main's schema). Those modules are not touched by this PR.

Additional Notes

• No base-branch fixup commits — main builds the parts this PR touches.
• Scope: only EventsService boot/poll flow, one new Prisma model +
  migration, and the health module. No changes to event handlers
  (handleCourseApproved, handleCertificateIssued, …), to
  StellarService, or to any other module.